This describes the process of building a new monitoring server from scratch. This was modeled on the HamWAN server running in our Fremont site, but build on a new vintage VM and with more modern software versions. Rather than Cacti, we use Zabbix as the monitoring engine.
Added Debian trixie net-install ISO to proxmox server
Create a VM, and install Debian trixie
Boot and configure as a minimal server using the graphical interface
apt install jq locate mg mtr postfix pylint python3-virtualenv redis-server redis-tools rsyslog strace sudo tcpdump
# update the locate database
updatedb
/sbin/groupadd hamadmin
add /etc/sudoers.d/hamadmin (copied from monitoring.hamwan.net)Change /etc/ssh/sshd_config to move the SSH server to port 222, then restart sshd with systemctl restart sshd.
My initial account had to be lower case (so kd7dk). I then fixed that to my standard HamWAN KD7DK account:
Note: at this point I believe our ansible automation is capable of creating all the other netop accounts.
The monitoring server sits on two networks and must never join them:
ens18): in the default VRFens19): in a VRF named mgmtWe use 2 routing domains (the default VRF and mgmt) to enforce separation. Why? One OSPF instance has one link-state database, so the host forms adjacencies on both networks and floods LSAs between them, and the two networks merge into one routing domain. Filters only control what is installed or redistributed, not what is flooded.
The design is:
ens19 lives in its own VRF (mgmt, kernel routing table 200), with its own OSPF instance and its own LSDB.The steps depend on each other. Do them in this order:
mgmt VRF and move ens19 into it (this section)mgmt interface in the host firewall (next section)Moving ens19 into a VRF flushes its addresses and routes, so do step 1 from the console, not over an SSH session that arrives via the management LAN.
This host uses classic ifupdown, which has no native vrf keyword (that is ifupdown2 syntax). The VRF is created from hooks in the ens19 stanza of /etc/network/interfaces. The address is added after pre-up, so it ends up inside the VRF.
# The management network interface
auto ens19
iface ens19 inet static
address 10.44.4.8/24
pre-up ip link add mgmt type vrf table 200 2>/dev/null || true
pre-up ip link set dev mgmt up
pre-up ip link set dev ens19 master mgmt
# only if mgmt needs a static default (omit if learned via OSPF):
#up ip route add vrf mgmt default via <mgmt-gw> dev ens19
# Policy routing: let host processes reach management hosts (see below).
up ip rule add to 10.44.0.0/16 lookup 200 pref 100
up ip rule add to 172.16.241.0/24 lookup main pref 90
# Loose reverse-path filtering; strict mode breaks VRF/NAT return paths.
up sysctl -w net.ipv4.conf.ens19.rp_filter=2
up sysctl -w net.ipv4.conf.mgmt.rp_filter=2
post-down ip rule del to 10.44.0.0/16 lookup 200 pref 100 || true
post-down ip rule del to 172.16.241.0/24 lookup main pref 90 || true
post-down ip link set dev ens19 nomaster || true
post-down ip link del mgmt || trueDo not use the gateway keyword here. It would place a default route in the main table. Management routes are learned by OSPF inside the VRF.
What the two ip rule lines do:
| Rule | Purpose |
|---|---|
pref 100: to 10.44.0.0/16 lookup 200 |
Processes in the default VRF (Zabbix pollers, the agent, ping) reach management hosts via the mgmt table. Without it, management prefixes exist only in table 200 and these processes can only use the public network. |
pref 90: to 172.16.241.0/24 lookup main |
Return path for Zabbix containers. Replies from management hosts arrive on the VRF and are addressed to the container (after NAT is undone), so the lookup for the Docker frontend subnet must use the main table, where the Docker bridge route lives. The subnet must match the pinned frontend subnet (see the Zabbix section). |
cat > /etc/sysctl.d/90-vrf.conf <<'EOF'
# Sockets in the default VRF accept traffic arriving on the mgmt VRF
net.ipv4.tcp_l3mdev_accept = 1
net.ipv4.udp_l3mdev_accept = 1
# Docker (Zabbix containers) needs forwarding
net.ipv4.ip_forward = 1
EOF
sysctl --systemDo not set net.ipv4.ip_forward = 0 anywhere in /etc/sysctl.d/. Docker requires it. Check for conflicting entries with:
grep -rn -E 'rp_filter|ip_forward|l3mdev' /etc/sysctl.conf /etc/sysctl.d/ /usr/lib/sysctl.d/The per-interface rp_filter settings are in the ens19 stanza above, because those interfaces do not exist when sysctl.d is read at boot.
ip -br link show master mgmt # should list ens19
ip -br addr show ens19 # 10.44.4.8/24
ip rule # pref 90 and pref 100 rules, then l3mdev at 1000
ip vrf exec mgmt ping -c2 <mgmt-router>A plain ping 10.44.4.x from the default VRF is not a valid test of the VRF itself. Use ip vrf exec mgmt.
The public network and the management network each get their own OSPF instance. The public instance runs in the default VRF; the management instance runs in the mgmt VRF. A single ospfd handles both, so the older approach of running two ospfd processes with instance numbers is no longer used.
For traffic received on a VRF slave interface, netfilter’s INPUT chain sees the input interface as the VRF device (mgmt), not ens19. Any existing rule that names ens19 must be changed to mgmt. Without this, no OSPF adjacency forms on the management network, even though everything else looks right.
iptables -I INPUT -i mgmt -p 89 -j ACCEPTOSPF is IP protocol 89 (multicast 224.0.0.5 and 224.0.0.6). Keep the rule scoped to mgmt so the public side does not accept OSPF from the wrong place. Persist it with whichever mechanism the server uses for its own host firewall rules (for example netfilter-persistent), and do not save Docker’s chains into the persistent ruleset.
apt install frrIn /etc/frr/daemons:
zebra=yes
ospfd=yesMake sure FRR starts after the network (and therefore the VRF) exists. On Debian this is normally already true, but check:
systemctl show frr -p After | tr ' ' '\n' | grep -E 'network|networking'If networking.service is not listed, add an override. Be sure to read the comments in the prototype provide:
systemctl edit frr
[Unit]
After=networking.service
Wants=networking.service/etc/frr/frr.conf:
frr defaults traditional
hostname monitoring.ziply.hamwan.net
log file /var/log/frr/frr.log
log syslog informational
no ipv6 forwarding
service integrated-vtysh-config
password <PASSWORD>
enable password <PASSWORD>
vrf mgmt
exit-vrf
! ---- public network (default VRF) ----
interface lo
ip ospf area 0
ip ospf passive
exit
interface ens18
ip ospf area 0
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 <OSPF_PASSWORD>
ip ospf priority 0
exit
router ospf
ospf router-id 44.25.67.58
max-metric router-lsa administrative
area 0 authentication message-digest
exit
! ---- management network (mgmt VRF, separate LSDB) ----
interface ens19 vrf mgmt
ip ospf area 0
ip ospf authentication null
ip ospf priority 0
exit
router ospf vrf mgmt
ospf router-id 10.44.4.8
max-metric router-lsa administrative
exit
line vty
!Notes:
redistribute connected, distribute-list, prefix-list and route-map filters are gone. The VRF boundary does the isolation. Do not add route leaking (import vrf) between the instances.max-metric router-lsa administrative advertises the host as a stub router so neighbours do not choose it as a transit path.mgmt VRF must exist in the kernel (step 1) before FRR loads this file, otherwise interface ens19 vrf mgmt and router ospf vrf mgmt are rejected. If FRR was started too early, run systemctl restart frr.systemctl enable --now frrvtysh -c 'show vrf'
vtysh -c 'show ip ospf neighbor' # public neighbours only
vtysh -c 'show ip ospf vrf mgmt neighbor' # management neighbours only
vtysh -c 'show ip route ospf' # public routes only
vtysh -c 'show ip route vrf mgmt ospf' # management routes only
ip route show vrf mgmt # kernel view of table 200
ip route get 10.44.4.1 # should leave via ens19 (policy rule)In show ip route, O>* means an OSPF route that is selected and installed in the kernel.
After a change, neighbours may take a few minutes to age out old LSAs.
Install git.
apt install gitinstall Docker according to https://docs.docker.com/engine/install/debian/ from https://github.com/zabbix/zabbix-docker/tree/7.2 Follow model docker-compose_v3_ubuntu_mysql_latest.yaml
Add the following to /etc/fstab: /dev/mapper/monitoring–data–vg-data /data ext4 defaults 0 2 followed by: systemctl daemon-reload
If necessary, mount /data.
\# install mysql (via mariadb fork)
apt install mariadb-server
systemctl stop mariadb
mkdir /data/mysql
chown mysql:mysql /data/mysqlEdit /etc/mysql/mariadb.conf.d/50-server.cnf datadir = /data/mysql innodb_buffer_pool_size = 8G
Add empty database to myql, grant access to ‘zabbix’ with a password.
mysql << DONE
create database zabbix;
grant all privileges on \*.\* to 'zabbix'@'localhost' identified by 'some-password';
DONECreate a user to run zabbix containers and install the agent:
useradd -m -c "Zabbix Monitoring" zabbix
# in the container zabbix-server
chmod u+s /usr/bin/fping
# Install zabbix-agent Debian package (here and on other Linux servers)
apt install zabbix-agent
# then configure /etc/zabbix/zabbix\_agentd.confServer=172.16.241.0/24
ServerActive=172.16.241.3:10051
Hostname=monitoring.ziply.hamwan.netThis uses the zabbix container addresses.
On any other repeater, this would changes would look likeL
Server=44.25.67.58
ServerActive=44.25.67.58:10051
Hostname=monitoring.ziply.hamwan.netServer can be an address, CIDR range, or list of either. See the manual for more details.
Disable unnecessary discovery in Mikrotik Template (CapsMAN, LTE)
The Zabbix server runs in a container on the zabbix-docker_frontend network, which carries its default route. Its polls to management hosts are routed through the mgmt VRF by the policy rule from the networking section and are NATed (masqueraded) to the host’s management address, 10.44.4.8. Several things must be in place for this to work and to survive reboots.
1. Pin the frontend subnet. The policy rule pref 90 in /etc/network/interfaces names the Docker frontend subnet. Docker assigns subnets automatically and may choose a different one if the network is recreated, so pin it in the compose file (merge into the existing frontend: network definition):
networks:
frontend:
ipam:
config:
- subnet: 172.16.241.0/24To check which network a container uses for its default route, read /proc/net/route inside it. The default route’s gateway is printed as little-endian hex (01F110AC is 172.16.241.1).
docker exec zabbix-docker-zabbix-server-1 cat /proc/net/route2. Do not bind-mount the MariaDB socket file. MariaDB runs on the host, with its data in /data/mysql. If a compose service has a short-syntax bind mount of the socket file, for example:
- /run/mysqld/mysqld.sock:/run/mysqld/mysqld.sockDocker creates a missing source path as a directory. If Docker starts the containers before MariaDB, /run/mysqld/mysqld.sock becomes a directory and MariaDB then fails to start with Bind on unix socket: Address already in use. Mount the directory instead, which always exists:
- /run/mysqld:/run/mysqldIf a directory has already been created at that path, stop the containers, remove it, and start MariaDB first:
docker compose stop
rmdir /run/mysqld/mysqld.sock
systemctl start mariadb
ls -l /run/mysqld/mysqld.sock # must start with "s"3. Start Docker after MariaDB. The directory mount above helps, but ordering Docker after MariaDB also removes the race:
systemctl edit docker
# [Unit]
# After=mariadb.service
# Wants=mariadb.service4. Allow replies from the management network through Docker’s anti-spoofing rule. Recent Docker versions add rules like the following in the raw table for each container:
-A PREROUTING -d 172.16.241.2/32 ! -i br-<id> -j DROPReplies from management hosts come back through a VRF slave interface. The kernel handles them twice in PREROUTING, and on the second pass the reply is addressed to the container but arrives on mgmt, not the Docker bridge. Docker’s rule drops it. The symptom is that the reply is visible in tcpdump on ens19 and on mgmt, but never on the bridge, and the raw PREROUTING DROP counter for the container increases.
Add a narrow exception ahead of Docker’s rules:
iptables -t raw -I PREROUTING 1 -i mgmt -s 10.44.0.0/16 -d 172.16.241.0/24 -j ACCEPTIt matches only the mgmt device, only management-network sources, and only the Zabbix frontend subnet.
5. Block transit from the public side into the management network. Because the pref 100 rule applies to all route lookups on the host, packets forwarded by this host that arrive on the public interface and are addressed to 10.44.0.0/16 would also be sent into the mgmt table. Block that explicitly, so isolation does not depend on neighbours never routing management prefixes to this host:
iptables -I DOCKER-USER -i ens18 -d 10.44.0.0/16 -j DROPTo test it, add a route on a host on the public segment (ip route add 10.44.4.1/32 via <ens18-ip>) and ping 10.44.4.1. The DROP counter in iptables -L DOCKER-USER -v -n should increase, and nothing from that host should appear in tcpdump -ni ens19. Remove the test route afterwards. Container polls and the host’s own traffic are unaffected, because they do not arrive on ens18.
6. Make both rules persistent. Docker re-creates its raw rules when it restarts or recreates networks, so do not save these rules in a saved ruleset. Re-add them from a systemd unit ordered after Docker:
# /etc/systemd/system/mgmt-raw-accept.service
[Unit]
Description=Allow mgmt VRF replies to Docker frontend containers (raw PREROUTING, transit guard)
After=docker.service
Requires=docker.service
PartOf=docker.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/sh -c 'iptables -t raw -C PREROUTING -i mgmt -s 10.44.0.0/16 -d 172.16.241.0/24 -j ACCEPT 2>/dev/null || iptables -t raw -I PREROUTING 1 -i mgmt -s 10.44.0.0/16 -d 172.16.241.0/24 -j ACCEPT'
ExecStart=/bin/sh -c 'iptables -C DOCKER-USER -i ens18 -d 10.44.0.0/16 -j DROP 2>/dev/null || iptables -I DOCKER-USER -i ens18 -d 10.44.0.0/16 -j DROP'
[Install]
WantedBy=multi-user.targetsystemctl daemon-reload
systemctl enable --now mgmt-raw-accept.servicePartOf=docker.service restarts the unit with Docker. If you recreate a compose network without restarting Docker (docker compose down then up), run systemctl restart mgmt-raw-accept afterwards.
7. Management hosts must accept the host’s address. Polls to management hosts now arrive from 10.44.4.8, not from the container address. SNMP ACLs and Zabbix agent Server= / ServerActive= settings on management-network devices must allow 10.44.4.8. The monitoring host’s own agent is unchanged (Server=172.16.241.0/24).
8. Verify from the container. Start the stack first (see Starting Zabbix under Zabbix Operations below), then verify from the container.
docker exec zabbix-docker-zabbix-server-1 ping -c2 10.44.4.1
docker exec zabbix-docker-zabbix-server-1 ping -c2 <host-in-10.44.32.8/29>
iptables -t raw -L PREROUTING -v -n | head -4 # counter on the mgmt ACCEPT rule increasesThen use “Execute now” on an item of a management-network host in the Zabbix UI and confirm it returns data.
Reboot once the whole stack is configured, and confirm everything comes back in the right order:
ip -br link show master mgmt # ens19 is in the mgmt VRF
ip rule # pref 90, pref 100, l3mdev at 1000
ip route show vrf mgmt # connected route plus OSPF routes
sysctl net.ipv4.tcp_l3mdev_accept net.ipv4.udp_l3mdev_accept net.ipv4.ip_forward
sysctl net.ipv4.conf.ens19.rp_filter net.ipv4.conf.mgmt.rp_filter # both 2
vtysh -c 'show ip ospf neighbor' # public neighbours only
vtysh -c 'show ip ospf vrf mgmt neighbor' # management neighbours only
iptables -S INPUT | grep mgmt # OSPF accept rule present
iptables -S DOCKER-USER # Check for transit guard rule
iptables -t raw -S PREROUTING | head -3 # mgmt accept rule above Docker's drops
ls -l /run/mysqld/mysqld.sock # starts with "s"
docker compose ps # Zabbix containers healthyIsolation check: show ip ospf neighbor must list only public neighbours and show ip ospf vrf mgmt neighbor only management neighbours. Neither routing table should contain prefixes from the other network.
| Symptom | Cause | Fix |
|---|---|---|
No OSPF neighbours on the management network; tcpdump shows hellos arriving |
INPUT firewall rule matches ens19, but INPUT sees the VRF device mgmt |
Match -i mgmt |
| Host cannot ping its own management address or management hosts from the default VRF | Management address and connected route live in table 200 only | Use ip vrf exec mgmt ..., or rely on the pref 100 policy rule |
| Host processes reach public hosts but not management hosts | Policy rule missing | Check ip rule for the to 10.44.0.0/16 lookup 200 rule |
| Services listening in the default VRF refuse connections arriving on the management side | VRF socket scoping | net.ipv4.tcp_l3mdev_accept=1 (and udp_l3mdev_accept=1) |
Container pings to management hosts time out; replies visible on ens19 and mgmt but not on the bridge |
Docker raw-table anti-spoofing rule drops replies arriving on mgmt |
Add the raw PREROUTING accept rule |
| Container pings still fail with replies reaching the bridge side | Missing pref 90 rule, or NAT not applied |
Check ip rule and iptables -t nat -L POSTROUTING -v -n |
MariaDB fails with Bind on unix socket: Address already in use and nothing else is running |
mysqld.sock is a directory created by a Docker bind mount |
Mount /run/mysqld as a directory, start MariaDB before Docker |
interface ens19 vrf mgmt rejected by FRR |
FRR started before the VRF existed | Create the VRF first, then systemctl restart frr |
| Zabbix cannot reach anything after Docker or iptables was reloaded | Docker chains removed by a ruleset restore | Restart Docker; keep Docker’s chains out of saved rulesets |
cd ~zabbix/zabbix-docker
docker compose -f psdr.yaml --profile all up -dcd ~zabbix/zabbix-docker
docker compose -f psdr.yaml --profile all downdocker logs -f _container_id_or_name_
If you want to change one of the Zabbix parameters that are specified in config files in /home/zabbix/zabbix-docker or a subdirectory like env_vars, you need to stop Zabbix and rebuild the effected containers.
cd ~zabbix/zabbix-docker
docker compose -f psdr.yaml --profile all down
docker compose -f psdr.yaml --profile all build --no-cache --parallel
docker compose -f psdr.yaml --profile all up -d‘all’ can be replaced with the specific container if you know the impact of the update is limited but the safest option is to use all.
Updated the Mikrotik by SNMP template to enhance dashboards and turn off some data collection (LTE, CAPSman)
Added Template for AirFiber by SNMP, and then created a varient with the PING and inventory update disabled/removed so that it can be combined tiw Network Generic Device by SNMP which provides important additional Triggers and Graphs not available in the AirFiber template as provided.
Upgrades come in 2 pieces
Basic steps:
To modify a parameter in env_vars/*:
For a minor upgrade (e.g. 7.4.10 -> 7.4.13):
For a major upgrade (e.g. 7.2 -> 7.4):
(Older instructions… This should probably be done more like the minor upgrade process.)
Used configuration from Fremont largely unchanged. There is a key dependency on the Unfiltered.log file where the bulk of HamWAN infrastructure logs. Hacking attempts are fed into fail2ban from here. Configuration of client logging needs review and it had lots of hardcoded 44.24 addresses that were never updated. Need to consider what kind of separation we actually need.
Basically out of the box. Will probably need to update to add authentication.
Used new fail2ban.conf as starting point and merged in HamWAN changes. Added new HamWAN files from Fremont with review.
These provide a redis server and the long poll web servers for new bans. https://github.com/kd7lxl/blacklist-service
You also need to add this to 000-default-ssl.conf for apache:
\<Location "/blacklist"\>
ProxyPass "http://127.0.0.1:1234/"
\</Location\>You will need to enable mods proxy and proxy_http:
Add /srv/www/keys and this stanza to 000-default-ssl.conf for apache:
<Directory /srv/www/keys/>
Options +Indexes +FollowSymLinks
AllowOverride None
Require all granted
</Directory>Use the scripts in HamWAN Github repository.
ICMP ping loss trigger is too sensitive. Needs a longer sample interval I believe.
redis is complaining at startup: WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add ‘vm.overcommit_memory = 1’ to /etc/sysctl.conf and then reboot or run the command ‘sysctl vm.overcommit_memory=1’ for this to take effect.
S2.CapitolPark.hamwan.net MikroTik: Interface wlan1(): Link down Caused because a sector that loses its last client connections will change OperStatus to down instead of dormant. This trigger may need to be revised for APs, or possibly deleted for APs.
zabbix-server-1 | 69:20250205:143203.601 cannot send list of active checks to “172.16.241.1”: host [monitoring] not found. This is a configuration issue I believe.
Figure out how to ensure all the proxmox servers have the seme inventory of install images.
Web GUI shows a valid certificate but complains about active content with certificate errors. Something like this posting: https://community.letsencrypt.org/t/chrome-69-0-3497-81-reports-active-content-with-certificate-errors/71545 Resolution: This was caused by cached javascript content from the prior self-signed certificate. Cleared the cached content and all was fine.
Ping health checks failing. Needed to add setuid bit to fping in the server container.
SNMP agent item “net.if.wireless.walk” on host “r1.capitolpark.hamwan.net” failed: first network error, wait for 15 seconds (and similar) Possibly: https://www.zabbix.com/forum/zabbix-troubleshooting-and-problems/483095-zabbix-7-snmp-timeouts Fixed this by increasing the timeout for SNMP queries (Administration > General > Timeouts). I changed it from 3s to 10s.
3 Mikrotik hosts are refusing to respond to SNMP get/getbulk In this case s2.indianola, r3.baldi and capitolpark.queenanne. This turned out to be a semi-known issue with SNMP and Mikrotik, and asymmetric routes. RouterOS would respond with the address of whichever interface had the best route back to the requester. If that was not the interface that the request was sent to, the requester would be unable to match it with the request it sent. Solution is to give all devices a stable address (on their loopback interface if they have more than one interface), make that the default address in the portal, and set src-address in /snmp to force all SNMP response to come from that address.
Mikrotik templates (from Zabbix and third parties) https://www.zabbix.com/integrations/mikrotik
https://www.zabbix.com/documentation/current/en/manual/discovery/network_discovery